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I. INTRODUCTION 



The University of Denver (DU) was awarded a 42-month contract from 
the U.S. Department of Education's Office of Special Education Program (OSEP) 
to develop computer-based simulations in the area of job-related social skills for 
learning handicapped youth, aged 16 to 21, who are in the transition between 
school and work. The official start date for this project was September 30, 1985, 
and the project was to be completed by March 29, 1989. For a variety of 
organizational, administrative, and fiscal reasons, however, the decision was made 
to close out the contract on tMarch 31, 1988. The purpose of this report is to 
provide (a) a historical account of all project activities from the beginning of the 
project up to the final activities; (b) a description of all products developed under 
the contract; (c) a discussion of evaluation data collected to date, along with a 
description of relevant evaluation procedures; and (d) recommendations regarding 
future implementations of the completed job-related social skills curriculum 
(SUCCESS program) and additional research and evaluation activities. 

Rationale and Objectives of the Project 

The U.S. Department of Education estimates that more than 300,000 
handicapped youth leave school annually. Of those that are in high school 
programs, it is further estimated that less than one-third receive instruction or 
training related to employment. In those existing programs for handicapped 
youth, most of the instructional time is concentrated on improving academic and 
vocational competencies, with less attention being paid to improving social 
competence related to gaining and retaining stable employment. Yet, research 
shows that frequently it is social competency that largely determines whether or 
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not handicapped persons are successful in seeking and retaining employment. 
Thus, the purpose of this project was to study the effectiveness of 
computer-based simulations as an approach to helping handicapped youth acquire, 
transfer, and generalize specific job-related social competencies. 

The project was initially planned to proceed through four stages: an 

analysis of critical employment situations and their social skills components, the 
design and development of computer-based simulations to teach the identified 
social competencies, the implementation and evaluation of simulation materials in 
school and work settings, and the dissemination and marketing of project results 
and products. These four stages roughly corresponded to the 4 years of the 
project. Due to marketing lead-time requirements of the publisher selected to 
manufacture and distribute the finished products, the dissemination and marketing 
activities, originally planned for the last year of the project, were moved up to 
year 2 and the evaluation activities were correspondingly moved to years 3 and 4. 

Population to be Studied 

The general population to be studied in this project was learning 
handicapped youth in the age range of 16 to 21 years. In particular, three 
subgroups within this population have been identified: the learning disabled (LD), 
the mildly retarded (MR), and the emotionally disturbed (ED). The population 
meeting these critera were to be found among secondary school students in the 
transition between school and work (e.g., juniors and seniors in high school). 
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II. HISTORICAL ACCOUNT OF PROJECT ACTIVITIES 



Key Project Activities and Events 

The following is a chronological description of key project activities and 
events, beginning with initial project activities in October of 1985, and ending 
with termination of the contract on March 31, 1988. 

A project kick-off meeting was held on October 2, 1985, during which 
key University of Denver/Denver Research Institute (DU/DRI) staff met to 
discuss schedule and activities for the first 3 months of the project. A staff 
meeting schedule was set up on a weekly basis, with meetings to be held each 
Wednesday. Contact was made with the Project Officer, Dr. Jane Hauser, and 
arrangements were made for a teleconference to go over project goals and 
reviewer comments. The teleconference was held on October 16 with all DU/DRI 
project staff in attendance. 

Per the request of the Project Officer, contacts were made with Louise 
Appell and Hellen Pollack of Macro Systems, Inc. to closely coordinate our 
respective efforts on this project. Project member. Dr. Joe Lamos, met with 
Louise Appell at the Closing the Gap conference on October 31. At that time 
proposals were exchanged and our respective approaches were discussed. It was 
agreed to maintain close contact throughout the project and to coordinate our 
efforts as much as possible. Differences in our approaches were discussed as well 
as potential ways to maximize the complementarity of our efforts. Contacts 
were also made with Patti Zembrosky-Barkin and Philip Browning at the 
University of Oregon who were involved with a related OSEP project (Project 
LIVE). 




3 



Project Consultant, Dr. Steve Hazel from the University of Kansas 
Institute for Research on Learning Disabilities (KU-IRLD), visited DU/DRI on 
October 29. During this visit, technical assistance was received on existing social 
skills curricula to be included in our review, simulation design considerations, 
effective dissemination strategies, and strategies for working effectively with 
Review Board members. 

Over 100 publishers and professional organizations that were identified 
as active in the area of education of special populations of relevance and/or social 
skills training curricula were contacted to supply us with their latest 
developments in these areas. Reviews of existing job-related social skills 
curricula were begun, along with analyses of critical employment-related social 
interactions and potential simulation approaches. 

During November of 1985 contacts were made with the three field sites 
(Adams County District 50, Jefferson County Secondary Schools, and Front Range 
Community College). Briefings were given on the project's goals, tasks, and 
support areas requested in terms of field site involvement during the project. 
Further points of contact were established at each site to provide infomation on 
exact nature of computer and video equipment that would be available for the 
computer-based simulations, as well as teachers and employers we can work with 
initially to provide further information on job-related social skill deficiencies in 
the three learning handicapped populations of interest (LD, ED, MR) and to 
identify students we could interview to gain a deeper understanding of their 
perceived job-related social skills problem areas. Also in November of 1985 
student, teacher, and employer interview forms were designed to be used in the 
collection of information on job-related social skills deficiences and critical social 
situations in employment settings. 
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Project consultant, Dr. Steve Hazel from KU-IRLD, visited DU/DRI on 
November 18-19. During his visit, additional information on existing social skills 
curricula was exchanged and skill training components identified. In addition, DU 
consultant. Dr. Susan Harter, met with the project team and Dr. Hazel to further 
refine the theoretical framework on which the computer-based simulations will be 
based. 

In December of 1983 interviews were scheduled and conducted with 
students, teachers, and employers associated with three of our selected field sites 
(Front Range Community College, Adams County District 50, Clayton 
Foundation). Information was collected regarding job-related social situations 
which are problematic for the student populations of interest (learning disabled, 
mildly retarded, emotionally disturbed), as well as social skill deficiencies related 
to these problem situations. This information was compiled as part of our analysis 
of critical employment-related social situations to be selected for computer- 
based simulation development. In addition, work continued on the review of 
existing social skills curricula and video materials and on the review of relevant 
literature on social skills training interventions and simulation technologies. 
Potential approaches to the training and simulation technology applications were 
documented as part of the analyses of feasible approaches. Also at this time a 
detailed application for conducting research at the fourth field site, 3efferson 
County Public Schools, v/as completed and submitted for District approval. 

In 3anuary 1986 approval was received for field site testing within the 
3efferson County Public School District. Further interviews with employers of 
mildly handicapped youth at three of the selected field sites were conducted to 
identify types of critical employment-related social situations that present 
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difficulties for the target population. These data were summarized to provide a 
basis for the design of an employer survey instrument that could be used to 
collect validation data for situations identified. Plans were established for 
conducting this survey in February with regional and nonregional employers of 
both handicapped and nonhandicapped youth in the transition from school to work. 

The first meeting of the project's Review Board was held on January 1^ 
with all selected Board members in attendance, along with project staff and 
project consultant Dr. Stevel Hazel of KU-IRLD. At this meeting the review 
board was updated on project status and agreement was reached on the 
preliminary categories of social skills/situations that are candidates for 
simulation development. These categories are: asking for help, giving help, 

accepting positive and negative feedback, giving positive and negative feedback, 
and expressing opposing points of view. Discussions with Review Board members 
were also conducted concerning technology and simulation delivery strategies. 
Review board members generally agreed that the technology chosen should be one 
that is most likely to be available within the next 3 years, as well as one that can 
make maximum use of simulation capabilities. The possibility of using animation 
versus live modeling was also discussed and it was agreed that animation offers 
some distinct advantages for the target population due to the ability to break skill 
components and situational cues down into easily discriminated and taught units. 
During the Review Board meeting, the opportunity was provided for individuals to 
indicate the task area (social skills, simulation technology, evaluation, 
dissemination) in which they would most like to be associated for the purpose of 
subsequent small working group meetings. 

Meetings were held with administrators and special education teachers 
from the three participating Jefferson County high schools on February 20. At 



that time they received a briefing on the project, progress to date, and 
preliminary design directions for the simulation materials* Individual contact 
persons at each of the schools were designated and follow-up interviews were 
conducted with students and teachers at each school. In addition, information on 
available technology for possible delivery of the simulation materials was 
collected. 

During February 1986 an employer survey form was designed for the 
collection of validation information concerning the classes of job-related social 
skills and employment situations identified as target areas for the simulations. 
The survey forms were sent to 19 employers of both handicapped and 
nonhandicapped youth in order to analyze the comparability of social skills 
problems for youth in general versus the three handicapped populations of interest 
in this project (see Design Format Report (Task 3), submitted to the OSEP on 
March 2^, 1986). 

A major portion of the general design for the five levels of simulation 
materials to be developed was completed during this time period. The details of 
the three computer-based levels were laid out in terms of the specific skill 
components to be addressed, the instructional strategies to be incorporated, and 
job situations to be addressed. A cast of characters to be used in the simulations 
was defined, along with the simulation approach and focus for each level. The 
Level U role-playing design and Level 5 on-the-job support training were also 
defined in terms of general content and strategies. 

Interviews with the Jefferson County field sites (teachers, students, and 
employers) were completed in early March of 1986. Discussions were also held at 
these sites with administrators to determine simulation technology alternatives 
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and their projections on the types of technology the schools would have available 
during the period of materials tryout and evaluation. 

The design report was completed on March 24 and sent to the OSEP for 
review. The report described the theoretical basis for the design of the 
simulation materials, the results of field site interviews and employer surveys, 
instructional design issues and the general design for the five levels of the 
simulation materials, hardware and software design considerations and 
recommendations, and evaluation issues to be addressed in the formative and 
summative evaluations. 

Drs. McCombs and Lamos of DRI met with Dr. Johnson and Hauser at 
the OSEP on April 24, 1986. Meeting topics included a review of the general 
design for the simulation materials and a discussion of the recommended 
microcomputer technology for delivery of the simulation materials. With respect 
to the recommended choice of microcomputer technology (i.e., Apple's Macintosh 
or IBM), the OSEP indicated, based on recent national surveys of the technology 
based in the schools, that the Apple lie computer was predominant and thus would 
have to be the technology of choice. It was agreed that in order not to 
compromise the agreed upon design, we could assume an Apple lie computer with 
128K of memory and two disk drives. It was also agreed that some add-ons to this 
basic system would be acceptable (e.g., to provide the necessary audio interface 
capabilities). In addition, because some modifications to the design might be 
required, that immediate testing out of various possibilities would begin to 
determine the extent of design revisions that might be necessary. 

On April 28, meetings were held with project consultants Drs. Steve 
Hazel and Susan Harter to review the OSEP meeting recommendations, discuss 
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hardware and software options to be tested, and plan the development of various 
types of evaluation measures (i.e., self-worth, job-related confidence and 
competence). The second meeting of the project’s Review Board was held in 
Denver at the University of Denver on April 29. The general purpose of the 
meeting was to discuss their comments and suggestions regarding the general 
design of the simulation materials, which members had reviewed prior to the 
meeting. There was also some discussion at the meeting of hardware and 
software issues and the expression of some concern that the product developed for 
Apple technology may not meet the needs of secondary level special education 3 
years from now, as members believed more advanced technology was likely to be 
available. Members were also generally of the opinion that special education 
programs were more likely to purchase a stand-alone package such as the 
Macintosh rather than purchase the add-ons for Apple II computers that they did 
not have dedicated access to at the secondary level. 

During May of 1986 detailed design specifications were completed for 
the two simulation strands (Accepting Criticism, Asking for Help) that are being 
developed under this contract. The detailed specifications included scenarios, 
diagnostic questions, discussions, and remediation sequences for the three on-line 
simulation levels; role playing cue cards and performance checklists for Level 
off-line support materials for employers, students, and teachers for Level 5; and 
off-line support materials for teachers for the first four simulation levels. 
Following telephone conversations with the contract monitor, it was decided that 
a representative sample of the detailed design for each level and simulation 
strand would be included in the Development Specifications Report to be 
submitted in early June. In addition, the report included the results of our 
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evaluation of several hardware and software options for executing the design on 
the Apple II family of computers. 

The Development Specifications Report detailing the design of 
representative samples of materials for each of the five levels of the social skills 
simulation program was completed and submitted to the contract monitor 
on June 9, 1986. During the month of June, the Formative Evaluation Plan report 
was also completed and submitted to the contract monitor on June 30. 

During this time, work also continued on evaluating various software 
development packages to support the design, including various digitized speech 
programs and graphics packages. It was determined during this process that 
available digitized speech packages for the Apple II family of computers did not 
meet the specifications of the materials design. Consequently, the decision was 
made that a more desirable approach was to develop an interface for the Apple II 
that will provide a computer-controlled. audio capability. A relatively inexpensive 
cassette deck (cost around $150) produced by Akai was located which would allow 
random access to audio tape sequences, thus supporting the degree of 
interactivity required. Through the interface to be developed by DRI, the 
computer will be able to search forward or backward to find and play selected 
segments of the tape. The interface will also tell the computer when the tape 
sequence is finished, and each segment of the tape would be indexed so that the 
computer would know which segment to locate and play. 

Via an ongoing publisher communication process, we identified three 
publishers of vocational materials for handicapped learners who were interested in 
working with us during our development process, with the long-term aim of 
potentially marketing the products developed as a single package. These 
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publishers were Harvest Educational Labs, DLM Teaching Resources, Hartley 
Publications, and Educational Activities Inc. Each of these publishers expressed a 
great deal of interest in the concept, the basic design, and the technology 
decisions made thus far. Each offered to provide technical advice and feedback 
on our evolving design, and expressed interest in reviewing our finished product 
for possible addition to their product lines. 

All storyboards for initial formative evaluation tryouts with field site 
teachers and students were completed in 3uly of 1986. In addition, plans were 
made to begin audio production on a small sample of scripts in September in order 
to finalize production procedures as well as produce a demonstration program that 
can be shown to the set of publishers who have expressed interest in the product. 
During this time period work had also progressed on the development of random 
access audio interface capabilities and it was expected that a prototype would be 
completed and tested by the middle of September. Work had also progressed on 
the detailing of all off-line support materials for students and teachers. 

During August of 1986 work continued on the development of detailed 
storyboards for the first set of simulation materials to be evaluated in the initial 
formative evaluations at our selected field sites during the Fall and Winter of 
1986. A professional artist was added to the staff for the purpose of taking initial 
rough drawings of all graphics and converting these to final artwork that would 
become the computer graphics. The audio production specialist had located 
professional narrators for the audio portions of the materials, and the first taping 
session was scheduled for the week of September 15. 

Mr. A1 Harris of Educational Activities, Inc., visited the project staff in 
Denver on August 29, at which time he was briefed on the product being 
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developed, the rationale for the approach chosen, and was shown a demonstration 
of samples of the simulation materials and the computer-controlled audio 
capability. He indicated that he was extremely interested in the product. We 
also continued to pursue our contacts at Harvest Educational Labs, DLM Teaching 
Resources, and Hartley Publications. A demonstration program was being put 

together for these publishers. In addition, during this time period work continued 

/ 

on the development of the computer-controlled random access audio interface. A 
prototype had been completed, and testing of the device was in progress. 

The project's Review Board met with project staff on September 9. The 
main purpose of this meeting was to go over their comments on scripts they had 
been sent to review. Project consultant. Dr. Steve Hazel, also attended this 
meeting and stayed through September 10 to work further on other project 
activities. Review board members had several helpful comments on the scripts 
and basicially felt they were appropriate to the. populations of interest, were 
presented at the right difficulty and reading levels, and taught needed skills. 
Some time at this meeting was devoted to discussing the potential marketability 
of the materials, with agreement reached that the final product should be helpful 
not only to the populations for which it is specifically intended in this project, but 
also to a wide range of young adults and adolescents. 

By October of 1986 taping of the first six prototype scripts had been 
completed and work was in progress on completing the development of the 
computer-based lessons that interfaced with these tapes. Additional interfaces 
were being built to support the tryouts of these lessons with students in the 
participating field sites. A tryout schedule for the six prototype lessons was 
established with the participating Adams County and Jefferson County high 
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schools. Work continued on the development of off-line student and teacher 
materials, as well as on the development of competence measures to be used later 
in the formative evaluation. 

The development of the computer-based portion of the first prototype 
lesson was completed the week of Novem.ber 24. This development was somewhat 
slower than anticipated due to a number of extra programming requirements (e.g., 
memory limitations, timing of graphic/audio segments). New software routines 
were systematized, however, which would aid in the subsequent development of 
the remaining prototype lessons. 

The project's Review Board met on December 2, at which time they were 
presented with a demonstration of a completed prototype lesson. Comments on 
this lesson were solicited, along with implementation suggestions. In general, the 
Board was extremely positive about the lesson and had no negative comments or 
suggested revisions. It was suggested that the lesson might be implemented such 
that groups of students could actively participate via programmed breaks 
combined with off-line exercises at logical points in the lesson. The Board also 
made a number of suggestions regarding the form and content of off-line teacher 
materials that could accompany the completed social skills curriculum. 

By December 1986 three of the six prototype lessons for the com uter- 
based portion of the job-related social skills program had been developed. The 
first of these was subjected to a tryout with students at the selected field sites 
during the first week in December. Students in the three handicapping conditions 
from three of the participating high schools were involved in this tryout of the 
first prototype Level 1 lesson. Results of this tryout generally indicated that 
students liked the lesson, found it interesting, and indicated that it was at the 
appropriate level of difficulty. 
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Responses had been received from a number of potential publishers. Mr. 
Bruce Brown with Media Materials in Baltimore set up a visit to see a 
demonstration of the developed prototype materials on January 15. During this 
meeting, he expressed considerable interest in publishing both the simulation 
materials and the interface device. All staff members were favorably impressed 
with Media Materials' marketing strategies and publication philosophies such that 
continued contact is being maintained. 

Two additional prototype lessons for the computer-based portion of the 
simulation materials were subjected to student tryouts at the Adams County and 
Jefferson County field sites the weeks of January 19 and 26, respectively. These 
were Level 2 prototype lessons from the Accepting Criticism and Asking for Help 
strands. As with the Level 1 prototype which was subjected to tryout in 
December, these tryouts indicated that students liked the lessons, found them 
interesting, and felt the materials were generally at the right level of difficulty. 

Work continued during January and February on the development of the 
remaining three prototype lessons and on five additional Level 1 and Level 2 
lessons. A refinement was made to the audio/computer interface device such that 
it was now possible to use both sides of the audio tape. This refinement reduced 
the amount of search time for individualized branching and generally improved 
the efficiency of the technology. 

The project's Review Board met on March 3, at which time the major 
agenda item was to solicit comments on the Level 2 and Level 3 prototypes. Dr. 
Steve Hazel from KU-IRLD also attended this meeting and worked with project 
staff on further delineating the off-line materials. Review Board comments on 
the prototype lessons were generally very positive, with a few minor suggestions. 



O 

ERIC 



14 



With respect to publisher contacts, three persons affiliated with the Hartley 
Publications visited the project on March 6 and spent the day viewing 
demonstrations and discussing publishing possibilities. Following their visit, the 
project staff reviewed all publisher contacts made to date and determined that 
Media Materials represented the best match in terms of publishing philosophy and 
marketing strategies. A formal agreement with Media Materials was then 
initiated. 

Additional tryouts were held the week of March 9 on the Level 3 
prototype at the Jefferson County field sites. A revision to Level 2 based on prior 
tryout data was completed and tryouts of this revised version were scheduled for 
the latter part of March. Production of remaining Level 1 and Level 3 lessons was 
also in progress. 

Also during March of 1987, meetings were held with Dr. Susan Harter to 
reach final agreements on the types of competence measures to be used in the 
final evaluation of the simulation materials. The format and structure of these 
measures were finalized and the types of items that still needed to be written 
were defined. Dr. Harter's available measures were being used, along with some 
additional measures that could more directly assess increases in students' 
perceptions of social competence. 

During April 1987 work continued on the development of both on-and 
off-line materials for the job-related social skills simulation program. 
Storyboards and scripts were completed for 23 of the 28 on-line lessons, and 
coding was in progress on these lessons. Work also continued on further refining 
the off-line materials for teachers and students. In particular, the Level 4 role 
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playing materials, their formats, and teacher support information were refined. 
These materials were then sent to Media Materials for review and input. 

The University's contracts personnel were engaged in finalizing 
contractual agreements with Media Materials for the simulation project products. 
Additional tryouts of several on-line lessons with mixed ethnic characters were 
planned to ensure that the content was appropriate. High school students from 
the Clayton Foundation field site were used for these tryouts. Student reactions 
indicated that the content was appropriate and only minor revisions were 
suggested. 

The initial version of all competency measures to be used in the final 
evaluation of the simulation program were completed during April as well. The 
measures were: global self-worth, domain-specific competence, job-specific 

competence in the accepting criticism and asking for help areas, domain-specific 
important measures, and domain-specific anxiety measures. 

By May of 1987 considerable progress had been made in finalizing 
agreements with Media Materials. They had set up a production and publishing 
schedule for the materials, such that approximately half of the total program 
could be offered in their fall 1987 catalog. 

Key project personnel, Drs. McCombs and Lamos and Ms. Brannan, 
attended the Council for Exceptional Children Invitational Research Symposium 
on Special Education Technology in Washington, D.C. on May 31-June 2. During 
the symposium, a presentation on the project was given with Macro Systems, and 
a demonstration of the simulation program was provided. Those attending the 
demonstration expressed considerable interest in the materials and technology. 
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The project's Review Board met on June 23, during which time members 
had an opportunity to further review newly completed on-line lessons as well as 
drafts of off-line materials that have been prepared for the teacher guide and 
student notebook. Comments were solicited on the "readiness" of these products 
for commercial publication later this fall. The Review Board also participated in 
discussions about the format and content of Level 3 materials. Project 
consultant, Dr. Steve Hazel from KU-IRLD, participated in this meeting and spent 
another day with project staff going over issues related to our pending contractual 
agreements with the selected publisher. Media Materials. 

With respect to progress with the finalization of publisher agreements, 
Mr. Bruce Brown, President of Media Materials, visited the project on July 9 to 
discuss a variety of issues related to the agreements and to materials packaging. 
Final agreements were reached and documents were prepared for signature. It 
was agreed that the simulation program would be announced in their fall catalog, 
with materials to be available in November of 1987 (Phase I) and March of 1988 
(Phase II). The phased schedule was planned to allow input from field tryouts 
conducted by project staff and from site demonstration projects to be coordinated 
by Media Materials. 

It was at this time that the OSEP was asked to approve a budget 
modification to redistribute remaining project monies among budget line items. 
Unanticipated development costs associated with the requirement to use Apple II 
computers, the lack of adequate existing audio techniques for this family of 
computers, and reductions in original costs for outside consulting services 
necessitated this budget reallocation. It was anticipated at that time that the 
reallocation of remaining funds to staff salaries for development activities would 
be sufficient to allow completion of all project activities. 
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Final drafts of all off-line materials were completed and sent to Media 
Materials for review in August. Comments from the publisher on these drafts 
were received, indicating that these drafts were very favorably received. On that 
basis, the publisher proceeded with developing versions for publication. Work 
continued on completing the on-line lessons that are part of the Phase I product 
that was to be delivered to the publisher by the end of September. In addition, 
modifications to the audio interface were made to enable it to operate with a 
different cassette deck which the publisher has selected in place of the original 
Akai deck which was no Ipnger being manufactured. The publisher was planning to 
demonstrate the simulation program (hardware and software components) at the 
Closing the Gap conference being held in Minneapolis beginning on October 20. 
By this time the Teacher's Manual had been completed and a draft was being 
prepared for publisher review and comment. 

In September of 1987 the Phase 1 pilot lessons were completed and 
delivered Media Materials, along with a finished copy of all off-line materials that 
accompanied these seven lessons. Work was continuing on the remaining Phase I 
lessons, six of which were delivered to Media Materials on October 30 and two of 
which were delivered by November 15. Final agreements had by now been 
obtained for both the hardware and software portions of the simulation program. 
These agreements were signed by appropriate representatives of the University of 
Denver, Media Materials, and in the case of the software, U.S. Department of 
Education. Work was also in progress on modifying the formative evaluation plan 
to reflect a field test of the Phase 1 materials in the second semester of this 
year's academic year. The plan also specified a summative evaluation of the 
entire simulation program in the fall semester of 1988. 
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In December 1987 remaining Phase I lessons for the SUCCESS curriculum 
were completed and delivered to Media Materials. Work continued on the 
development of Phase II lessons, which were in various stages of programming, 
scripting, and storyboarding. During this time, plans were coordinated with the 
Jefferson County field site in Golden for a formative evaluation of Phase I 
materials beginning in late January 1988 and continuing through May 1988. It was 
at this time that a request for additional money to complete the project was 
submitted to the OSEP. Problems experienced with unanticipated changes in 
programming staff which led to increased numbers of programming hours was a 
primary factor in the need for additional funds. Furthermore, the imminent 
closing of the Social Systems Division of DRI had decreased normal administration 
support available to project staff. It was estimated that by the end of January 
1988, there would be nothing left of the original budget allocation for the project. 

Work continued on the Phase II lessons in January 1988, with a projected 
completion date of March 1988. Work was also proceeding on the off-line 
materials to accompany these lessons. In addition, plans were finalized for the 
field test of Phase I materials with Golden High School students in February. This 
evaluation was begun on February 3, 1988 with the training of four teachers and 
three administrators. These individuals participated in a half-day training session 
in which they were introduced to the background and philosophy of the 
curriculum, received a demonstration and practice with the on-line and off-line 
Phase I materials, and were briefed on the purpose and procedures of the 
evaluation. 

As of the end of March, 12 learning disabled, nine mildly retarded, and 
nine emotionally disturbed students were participating in combinations of 
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individual and group administrations of the Phase I materials. Specifically, seven 
of the learning disabled students participated in the group mode and five 
participated in an individualized mode; all nine mildly retarded students 
participated in an individualized mode; and the nine emotionally disturbed 
students participated in a group mode. Students in the group mode had completed 
nine of the 15 on-line lessons (i.e., the three introductory lessons, two Level 1 
lessons each in Accepting Criticism and Asking for Help, and two Level 2 lessons 
in Accepting Criticism). Students participating in the individual mode had 
completed fewer of these lessons, with the mildly retarded students completing 
the fewest number of lessons. 

During the December 1987 through March 1988 time span. University of 
Denver personnel were engaged in a series of information-gathering and 
negotiation sessions with Department of Education staff concerning the financial 
condition of this project. It was finally decided to terminate the project as of 
March 31, 1988. The project staff agreed to complete and deliver all on-line and 
off-line curriculum materials to the publisher and to the OSEP by that time, and 
the OSEP agreed to cover a cost overrun incurred up until that time. All 
curriculum materials were completed and delivered as agreed. This final report 
constitutes the final deliverable under this project. 
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III. DESCRIPTION OF PRODUCTS DEVELOPED 



Out of our review of the literature, available social skills training; 
products, interview and observation data. Review Board input and technology 
alternatives grew the design concept for the SUCCESS program. This program 
provides a unique blend of computer-based and print materials that lead the 
student through five progressive stages or levels of skill development, each of 
which helps prepare the student for behaving appropriately in social interaction 
situations with supervisors and co-workers. Both the computer-based and print 
materials are evenly divided between two major skill areas — Accepting Criticism 
and Asking for Help. The combination of on- and off-line materials in the 
SUCCESS program comprises a complete curriculum, designed to be presented in 
a 12 to 15 week school semester. 

The computer-based simulations that comprise the first three levels of 
skill development combine the realism of audiotaped voices with the animated 
characterizations of supervisors, workers, and co-workers interacting in job- 
related social situations. The combination of audio and graphic components in the 
simulations makes them particularly suited for students with reading difficulties^ 
The fourth level of skill development involves structured role-playing activities 
that give students the opportunity to apply newly acquired skills in realistic job 
situations, while also learning perspective-taking by taking on the roles of 
supervisors and co-workers. Skill development in the fifth level consists of 
opportunities for students to apply their skills in the actual job context, return to 
the classroom setting, and learn further techniques for forming support systems 
with their fellow workers and supervisors. 
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The computer-based portion of the SUCCESS Program consists of three 
introductory lessons and 2^ "simulation" lessons. These lessons have been 
designed in a sequence that begins with an introduction to basic social skills and 
then provides decision-making practice for the student in increasingly complex 
simulated job-related social situations. As students proceed through the lesson 
sequence, they move from the role of an observer to that of a participant in the 
simulated social situations. 

In addition to the computer-based lessons, the SUCCESS Program 
incorporates a variety of print materials, including tip sheets, lesson guides, and 
worksheets, to be used in conjunction with the computer-based lessons. They also 
include role-playing materials which form the fourth level of skill development 
within the program. The print materials have also been developed with the 
problem reader in mind. 

The following sections describe the on-line and off-line components of 
the SUCCESS Program in detail. 

On-line (Computer-Based) Products 

Introductory Lesson Materials 

The introductory lessons of the SUCCESS program are intended to 
introduce students to what good job-related social skills are and why they are 
important. Job-related social skill components are discussed within the context 
of asking for help and accepting criticism, and the NOTICE, THINK, SAY, and DO 
problem-solving model for job-related social behavior is introduced and explained. 
As the students proceed through the three introductory lessons, they are exposed 
to the basic skill components associated with successful job-related social 
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behavior, and are provided frequent opportunities to practice what they've learned 
by responding to questions. Introductory Lesson #1 teaches the skill components 
of NOTICE and THINK; Introductory Lesson #2 teaches the SAY skill components; 
and Introductory Lesson #3 teaches the DO skill components. The following is a 
description of each of the three introductory lessons. 

Title: Introductory Lesson #1: The NOTICE and THINK 

Skills 

Social Skill Components: NOTICE and THINK 

Instructional Strategy . Tutorial presentation. 3ohn, the narrator and 
"mentor" character who is used throughout SUCCESS is introduced. He begins by 
talking about what good job-related social skills are and why they are important. 
He then introduces the NOTICE, THINK, SAY and DO model and how it relates to 
accepting criticism and asking for help situations. The focus then shifts to a 
detailed treatment of the NOTICE and THINK skills. 

After the description and explanation of each skill component, the 
student is presented with a series of positive and negative examples of the skill 
component being applied in a social situation. These examples are followed by 
questions which allow the student to assess his or her own understanding of each 
skill component. Informative feedback is provided for both correct and incorrect 
answers to the questions. 

Instructional Content . For the NOTICE component, a student is 
instructed to notice certain things about others, including facial expression, what 
they are doing, tone of voice, and body language. The student is also told to 
notice his or her own feelings, thoughts, tone of voice, and body langauge. For 
the THINK component, the student is told to think about what they noticed in 
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others, what they have noticed about themselves, their goals, how the situation 
can be handled, and the consequences of different courses of action. 

Title: Introductory Lesson //2: The SAY Skills 

Social Skill Component: SAY 

Instructional Strategy . The instructional strategy for this lesson is 
basically the same as for Introductory Lesson //I. However, special emphasis is 
given to helping the student to discriminate between good talking and listening 
behaviors and those which are inappropriate. This is accomplished by an extensive 
series of short simulated job-related social interactions between a supervisor 
character and a worker character. Each example is followed by a question. 
Informative feedback is provided for both correct and incorrect student answers. 

Instructional Content . This lesson explains the SAY skill component. 
The student is told that the SAY skills encompass good listening and good talking 
skills. Good listening skills involve paying attention, repeating back what you 
have heard, and asking questions to get more information. The student is also told 
that good talking skills involve avoidance of the "you" message that implies blame 
or criticism, use of the "I" message to convey acceptance of responsibility, 
speaking assertively to let others know what you want, and talking politely to 
show respect for the other person. 

Title: Introductory Lesson //3: The DO Skills 

Social Skill Component: DO 

Instructional Strategy . This is a tutorial lesson. The instructional 
strategy for this lesson is the same as for Introductory Lessons //I and //2. This 
lesson focuses on how the student should appear and sound when engaging in a job- 
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related social interaction. The animated graphics of characters allow the student 
to identify appropriate facial expressions and body language while the audio 
component of the lesson allows the student to discriminate between the 
appropriate and inappropriate use of speaking skills. 

Instructional Content . The student is told that the DO skills include (1) 
maintaining a positive facial expression, (2) using a positive tone of voice, (3) 
using positive body language, and (4) doing what you said you would do. 
Maintaining a positive facial expression involves such things as making eye 
contact with the other person, having a calm or serious facial expression, smiling, 
etc. Using your voice positively involves not talking too loudly, speaking clearly, 
and speaking calmly. Using positive body language involves standing or sitting 
straight, using positive gestures, and not acting nervous. The student is told that 
doing what you said you would do is important to show that you really care about 
performing your job. well. 

The Level 1 Lessons 

General Description . Level 1 lessons in the SUCCESS program have been 
designed to help students begin to acquire the basic skills necessary for 
appropriate job-related social behavior in a nonthreatening setting. Through the 
Level 1 lessons the students are exposed to a variety of social problem-solving 
situations and are given the opportunity to make choices about how the worker in 
the situation should behave. 

Instructional Strategy . Each of the Level 1 computer lessons begins with 
the narrator describing a job-related situation, including the setting, characters, 
and a problem that must be resolved in some way. The students watch the worker 
and co-worker or supervisor act out the situation to a critical point at which the 
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worker must decide how to proceed. At that point the student is asked to choose 
from two possible courses of action for the worker in the situation. If the student 
makes the correct choice, a positive resolution of the social situation is played 
out by the characters in the lesson. If the student's choice is inappropriate, a 
negative resolution of the situation is shown. In the case where the student makes 
the correct choice, he/she is also shown the consequences of the incorrect choice. 
Likewise, when the incorrect choice is made, first the negative and then the 
positive situation outcomes are shown. This is followed by instruction that 
focuses on the positive outcome and the behaviors that contributed to it. The 
narrator's explanation of what happened in the situation is structured around the 
the NOTICE, THINK, SAY and DO model. The student is told what the main 
character noticed, thought, said, and did that resulted in a positive outcome. 
He/she is also told what the worker did incorrectly that resulted in the negative 
outcome. After the situation outcomes are presented and discussed, the student 
is given eight questions about what happened in the lesson. Correct and incorrect 
student responses to the questions are followed by informative feedback. 

Instructional Content . The specific instructional content varies from 
one lesson to the next. The lesson guide that accompanies each lesson provides 
specific information about basic skill components and specific skills addressed in 
each lesson. An important feature of the Level 1 lessons is that they provide the 
student with a clear contrast between appropriate and inapppropriate skill use for 
solving the problem posed in each lesson. The Level 1 lessons also provide the 
student with the ability to closely examine the consequences of applying each of 
the general NOTICE, THINK, SAY and DO skills. 

Level 1 Lesson Materials . Following is a list of the Level 1 lessons. The 
lesson guides that accompany each Level 1 lesson provide more detailed 
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information about the lessons, including a detailed synopsis of the lesson, 
information about lesson running time, skill components addressed, additional 
group and individual activities, and suggested answers for the student lesson 
worksheet. 

Level 1 - Accepting Criticism Lessons 
Title: Slow Sam 

Setting: Fast food restaurant 



Situation: 


Sam has been criticized for working too 
slowly. 



Characters: Sam and his supervisor 

Title: Sam and the Sandwich 
Setting: Fast food restaurant 



Situation: 


Sam serves his own customer a sandwich 
that had been prepared for his co-worker's 
customer. 


Characters: 


Sam and his co-worker 



Title: Chatty Sara 
Setting: A donut shop 



Situation: 


Sara's co-worker has criticized her for 
talking too much. 


Characters: 


Sara and her co-worker 



Title: Sara is Late Again 
Setting: A donut shop 



Situation: 


Sara is late for work for the third time. 
She's got to control her own negative 
emotions so she can deal with her 
supervisor's criticism in a positive way. 


Characters: 


Sara and her supervisor 



Level 1 - Asking for Help Lessons 
Title; Maria Asks the Dishwasher for Help 
Setting; A restaurant 



Situation; 


The restaurant is very busy and Maria, 
the bus person has just run out of clean 
glasses to reset the tables. 


Characters: 


Maria and her co-worker 



Title: Maria Sets the Tables 
Setting; A restaurant 



Situation: 


Maria doesn*t know how to set the tables 
for a special event 


Characters: 


Maria and her supervisor 



Title: Jackson Finds Out What To Do Next 
Setting; A landscape crew 



Situation; 


Jackson completes one task and has to 
find out what his supervisor wants him to 
to do next. 


Characters: 


Jackson and his supervisor 



Title: Jackson Learns from Matt 
Setting; A landscape crew 



Situation; 


Jackson is having a difficult time laying 
sod even though his co-worker has shown 
him how to do it. 



Characters: 



Jackson and his co-worker 



The Level 2 Lessons 



General Description . The Level 2 SUCCESS lessons have been designed 
to expose the student to a variety of ways for a worker to respond to a problem 
situation on the job. The possible solutions range from very negative to very 
positive. The intent is for the student to gain a better appreciation of the fact 
that any given problem can have more than one solution, but that not all of those 
solutions are good ones. 

Instructional Strategy . Like the Level 1 lessons, the Level 2 lessons 
begin with the narrator describing a job-related social situation involving a worker 
and a supervisor or a co-worker. At a key decision point in the interaction 
between the main characters, the student is presented vyith four possible ways the 
worker could resolve the situation. The student is then shown what could happen 
as the worker chooses each of the four options, and the narrator provides 
informative feedback about what happened. After each option is played out, the 
student is asked a question about what the worker did correctly or incorrectly in 
the situation. The narrator provides instructional feedback to both correct and 
incorrect student answers to the questions, referring to relevant parts of the 
NOTICE, THINK, SAY and DO Model. After all four options have been viewed, 
the narrator reviews the skill steps the worker used to handle the situation in the 
most positive way. 

Instructional Content . The specific instructional content varies from 
one lesson to the next. The lesson guide that accompanies each lesson rovides 
specific information about basic skill components and specific skills addressed in 
each lesson. The important feature of the Level 2 lessons is that the student can 
explore in detail a range of correct and incorrect ways for the worker to handle a 
single job-related social situation. 
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Level 2 Lesson Materials . Following is a list of the Level 2 lessons and a 



brief description of their content. A more complete description of each lesson, 
including a detailed synopsis, information about lesson running time, skill 
components addressed, additional group and individual activities, and suggested 
answers for the student lesson worksheet is provided as part of the lesson guide 
that is included with each Level 2 lesson package. 

Level 2 - Accepting Criticism Lessons 

Title: Careless Sara 

Setting: Donut shop 

Situation: Sara is careless and splashes her 

supervisor with hot oil. 

Characters: Sara and her supervisor 

Title: Billy's New Supervisor 

Setting: An office building 

Situation: Billy has a new supervisor with different 

work rules. 

Characters: Billy and his supervisor 

Title: Sam Makes a Bad Decision 

Setting: Fast food restaurant 

Situation: Many of the customers have to wait a 

long time for their food because one of 
the cooks did not come to work. Sam 
decides to give away free french fries to 
keep them happy. 

Characters: Sam and his co-worker 
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Title: Sue Wastes Time 



Setting: Department store 



Situation: 


Sue and Emily are supposed to hang up 
two racks of new clothing before the 
store opens. Sue spends most of her time 
putting together new outfits for herself 
instead of working. 



Characters: Sue and her co-worker 

Level 2 - Asking for Help Lessons 

Title: Maria and the Oyster Forks 
Setting: A restaurant 



Situation: 


Maria's co-worker asks her to put some 
special forks on the table for his 
customers. Maria doesn't know where 
they are kept. 


Characters: 


Maria, her co-worker, and her supervisor 



Title: Jackson and the Mower 
Setting: Landscaping crew 



Situation: 


Jackson takes the grass catcher off the 
mower and doesn't know how to put it 
back on. 


Characters: 


Jackson and his supervisor 



Title: Amy and the Special Order 
Setting: A department store 



Situation: 


Amy is having difficulty answering a 
customer's question. 



Characters: Amy, her customer, and her supervisor 



Title; Jose Doesn't Know What To Do 



Setting: Auto repair shop 

Situation: Jose needs specific instructions about how 

to perform a task. 

Characters: Jose and his co-worker 
The Level 3 Lessons 

General Description . The Level 3 lessons take students one step closer 
to actually performing in simulated job-related social situations by requiring them 
to assume the role of the main character in the lesson. This main character is 
required to respond to a complex social situation by choosing appropriate things to 
NOTICE, THINK, SAY and DO. 

Instructional Strategy . Each lesson begins with the narrator describing 
the setting, the characters, and the social situation involving, once again, an 
interaction between either a worker and a supervisor or a co-worker. After the 
context of the social situation has been shown via animated graphics and the audio 
tape, the narrator presents the student with a list of choices for things to NOTICE 
about the situation. The student is not allowed to proceed until the correct 
choice is made. Each incorrect choice is followed by informative feedback about 
why that choice was not the best. After the student succeeds in choosing the 
correct thing to NOTICE, he/she is shown a list of choices for things to THINK. 
Again, the student is not allowed to proceed until the correct choice is made. 

Immediately following the NOTICE and THINK choices, the student is 
presented with options about what to SAY in the given situation. After making a 
selection, the student is then presented with options for things to DO in the given 
situation. Based on the combination of the student's SAY and DO choices, a 
variety of outcomes will unfold for the student. 
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In the SAY and DO phase of the lesson it is possible for the student to 
SAY the right thing but DO the wrong thing, or vice versa. This has a decided 
impact on the outcome of the situation, and reinforces the important relationship 
between the SAY and DO skills. If the student makes an incorrect choice for SAY 
and/or DO, the narrator first shows the outcome of those incorrect choices, 
explains what went wrong, and then shows the outcome for the correct choices. If 
the student makes both an appropriate SAY and DO choice, the narrator shows the 
appropriate outcome, congratulates the student, and explains what went right in 
the situation. The narrator then continues with a review of the skills that were 
used to handle the situation appropriately. 

Level 3 incorporates an important element of "reality training" as well. 
One Accepting Criticism and one Asking for Help Level 3 lesson puts the student 
in the position of experiencing a negative outcome, regardless of the choices 
made. The narrator then explains that sometimes the student won't be able to 
achieve a positive outcome in a social situation because of factors outside his/her 
control. 

Instructional Content . The soecific instructional content varies from 
one lesson to the next. The lesson guide that accompanies each lesson provides 
specific information about basic skill components and specific skills addressed in 
each lesson. An important feature of the Level 3 lessons is that they provide the 
student with "real-life" practice in social situations, and insight into the 
interrelationships among the NOTICE, THINK, SAY and DO skill components with 
regard to determining situation outcomes. 

Level 3 Lesson Materials . Following is a list of the Level 3 lessons. The 
lesson guide that accompanies each lesson provides specific information about 
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lesson objectives, lesson running time, skill components covered, a detailed 
synopsis of the lesson content, additional group and individual activities, and 
suggested answers for the student lesson worksheet. 



Level 3 - Accepting Criticism Lessons 



Title; Sam Leaves Early 

Setting; Fast food restaurant 

Situation; Sam leaves work early to keep an 
appointment. He told his supervisor about it 
weeks ago but his supervisor has forgotten. 

Characters; Sam and his supervisor 



Title; Sara's Long Break 
Setting; Donut shop 

Situation; Sara goes shopping on her break and is late 
getting back. Her co-worker is very 
unhappy about this. 

Characters; Sara, her co-worker, and her super-visor 



Title; Sue's Embarrassing Problem 
Setting; Department store 

Situation; Sue neglects to bathe and wear clean 
clothing to work. Her co-worker notices 
this and talks to her about it. 

Characters; Sue and her co-worker 



Title; Billy's Messy Closet 
Setting; Cleaning crew 

Situation; Billy's storage closet at work is a mess and 
his supervisor criticizes him for it. 




34 



38 



Characters: Billy and his supervisor 

Please note: In this situation the student experiences a 

negative outcome regardless of the choices 
made. 

Lesson 3 - Asking for Help Lessons 

Title: Maria Can't Finish On Time 

f 

Setting: A restaurant 

Situation: Maria knows she won't be able to get all the 

tables set in time for the dinner customers 
unless someone helps her. 

Characters: Maria, her co-worker, and her supervisor 

Title: Jackson Waters the Lawn 

Setting: A landscaping crew 

Situation: Jackson needs help in figuring out how to 

turn on a sprinkler system. 

Characters: Jackson and his .supervisor 

Title: Jose's Confusing Instructions 

Setting: Auto repair shop 

Situation: Jose has a two conflicting sets of 

instructions for doing a task. 

Characters: Jose and his supervisor 

Please note: In this situation the student experiences a 

negative outcome regardless of the choices 
made. 

Title: Amy Makes a Suggestion 

Setting: A department store 
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Situation: 



Amy thinks there is a better way to do her 
work but isn't sure if she should make a 
change in the work procedure. 

Characters: Amy and her supervisor 
Off-line (Printed) Materials 

In addition to the on-line lessons already described, the SUCCESS 
Program incorporates a significant amount of off-line, or printed materials. 
These include the Level 4 role-playing material, the Level 5 on the job support 
material, a variety of worksheets, tip sheets and lesson guides, and the Teacher's 
Manual. Each of these printed products will be described below. 

The Level 4 Role-Playing Material 

General Description . Level 4 of the SUCCESS Program consists of 28 
role-playing situations, 14 for Accepting Criticism and 14 for Asking for Help. 
Each of the 28 situations involves a specific job-related social interaction 
problem. Level 4 departs in format from the previous levels in that there are no 
computer-based lessons. In addition to the role-playing material, evaluation 
checklists are provided for teacher use in assessing student performance in the 
role-playing exercises. Level 4 is intended to bring students one step- closer to 
actual, unprompted performance in difficult social situations. As such, it is an 
important part of helping students transfer and generalize their newly acquired 
social skills knowledge to novel situations and settings. 

Instructional Strategy . Role-playing is the primary strategy employed at 
Level 4. Through participation in the role-playing exercises and observation of 
other students doing role-plays, the student has a valuable opportunity to consider 
a problem from several points of view. By encouraging the students to play 
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through a situation and then switch roles and play it again, the teacher can 
provide useful training in impromptu problem solving and perspective taking. The 
role-playing activities also can serve as the basis for later group discussion. 

Instructional Content . The actual content of each role-playing situation 
is different, and depending on individual student interpretation of roles and 
situations, the skill usage in any given situation can vary from one exercise to the 
next. The evaluation checklists that accompany each role-playing situation 
provide guidance to the teacher about basic social skills that will contribute to a 
positive outcome for the role-play situation, but they are by no means intended to 
limit the student's solutions to the situations. 

Level 4 Role-Playing Material . The role-playing material consists of 28 
separate situations. Each situation has a worker card that describes the situation 
from the worker's point of view, a supervisor or co-worker card that describes the 
situation from that character's point of view, and a teacher evaluation checklist. 
Additionally, the teacher receives a set of general instructions for choosing 
students for the role-playing exercises, setting the stage, involving the rest of the 
class, evaluating student performance, and remediating students who have 
difficulty. 

The Level 5 On-the-Job Support Material 

General Description . At Level 5 of the SUCCESS Program, it is assumed 
that students are employed in part-time jobs, typically as part of a work-study 
experience. Thus, materials at this level are intended to provide the student with 
additional sources of support in their transition from the classroom context to the 
world of work. These sources of support are in the form of (1) encouraging an 
increasing independence and personal responsibility in using newly acquired skills 
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and (2) helping students tap into the sources of support that are available in the 
work environment. Level 5 is the final stage in the SUCCESS program's evolution 
from knowledge acquisition to skill development, to real-world transfer and 
generalization of job-related social skills. 

Instructional Strategy . The primary strategy in Level 5 involves group 
interaction, problem-solving, and sharing of actual job experiences. In addition to 
having students assume the primary responsibility for defining their specific job- 
related social situations that will be shared in the group, the group itself is 
encouraged to engage in generating alternative solutions. In this way, students 
participate in the "experience" of being responsible for solving their own problems 
as well as the "experience" of providing sources of support to each other. 
Students are also encouraged to share positive "success" experiences in accepting 
criticism or asking for help on the job and the strategies they used that 
contributed to the success of these experiences. The students can then construct 
their own "tip sheets" based on these shared experiences. These group activities 
are followed up by teacher-led structured discussions of techniques for building 
and effectively using social supports available on the job (e.g., other workers, 
supervisors, personnel counselors, etc.). 

Instructional Content . The content at Level 5 will necessarily vary from 
session to session, depending on the nature of the group and individual experiences 
being shared. In general, however, the content will be in the areas of Accepting 
Criticism and Asking for Help, and forming support systems on the job. The 
NOTICE, THINK, SAY and DO model will continue to be the primary "organizer" 
for structuring student discussions of job experiences. 
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Level 5 Materials . Materials that support the Level 5 activities consist 
of the Diary Worksheets, Employer Evaluation Worksheets, Student Group 
Problem Solving Tip Sheets, and Personal Networking Worksheets that students 
use to record the jpb experiences to be shared in the group. In addition, the 
teacher receives a set of general and specific instructions for structuring the 
group sharing/problem solving and for leading the follow-up discussion of 
strategies for building and effectively using available on-the-job support systems. 
These later materials, described in the Level 5 teacher instruction sheets, discuss 
techniques such as networking, making friends, and identifying those on the job 
who are appropriate sources of support and models for emulation. 

To assist in successful implemention the Level 5 activities, a variety of 
special teacher materials have been prepared for this level of the SUCCESS 
program. These teacher materials consist of: 

• Level 5 - General Teacher Instructions 

• Level 5 - Teacher Instructions for Sequencing Activities 

• Level 5 - Teacher Example: Group Problem Solving Tip Sheet 

• Level 5 - Teacher Example: Personal Networking Worksheet 

Lesson Guides, Tip Sheets, and Worksheets 

There are three types of printed material that accompany each one of 
the computer-based SUCCESS lessons: lesson guides, tip sheets and student 

worksheets. The lesson guide that accompanies each lesson provides specific 
information about lesson objectives, lesson running time, skill components 
covered, and a detailed synopsis of the lesson content. The tip sheet summarizes 
and generalizes the social skills and social context portrayed in each lesson. It 
can be used by the students as an aide to remembering how the worker handled 
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the situation portrayed in the lesson. The student worksheet asks the student a 
series of open-ended questions about the social situation portrayed in the lesson 
and how the main character in the lesson correctly responded to it. 

In addition to the student worksheets that accompany each SUCCESS 
lesson, five other kinds of worksheets were developed. These are the Diary 
Worksheet, Personal Networking Worksheet, It Happened To Me — Asking for Help 
Worksheet, It Happened To Me— Accepting Criticism Worksheet, and the Employer 
Evaluation form. These worksheets are intended to assist the student in relating 
the SUCCESS curriculum material to real-life events and problems. The "Diary" 
and "It Happened To Me" worksheets consist of a series of open-ended questions 
that guide the student through a description of a real-life event that relates to 
the curriculum material. After describing the event, the students are then led 
through a process of analyzing what went right or wrong in the situation, using the 
SUCCESS model, and then proposing alternative, more positive ways of solving 
similar situations in the future. The Employer Evaluation form is used to solicit 
information from employers about a particular student's performance on the job, 
or, as a less-threatening alternative, how an "ideal" employee would perform on 
the job. The Personal Networking Worksheet guides the student through a process 
of identifying appropriate role models or mentors on the job and making effective 
social contacts with that person. 

The Teacher Manual 

A 56-page Teacher Manual was prepared as a guide to the philosophy, 
content, and use of the SUCCESS Program. It has been organized into nine major 
sections, Section I, "Getting to Know SUCCESS," provides an overview of what is 
contained in the SUCCESS Program. Section II, "Understanding SUCCESS," 
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provides background information on the design, rationale, and use of the SUCCESS 
materials. Both Sections I and II are written using a question-answer format. The 
questions represent typical questions teachers have asked about the program and 
its materials. Section III, "Getting Started with SUCCESS," provides detailed 
procedural steps for the operation and use of the computer and audio equipment 
necessary to present the computer-based lessons. Sections IV through XI describe 
the format and use of each of the major levels of program material. 
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IV. EVALUATION PROCEDURES AND DATA 



The evaluation originally planned for this project included both 
formative and summative phases. The formative evaluation was intended to 
address key design and implementation issues relevant to the implementation of 
the SUCCESS curriculum in school settings. The summative evaluation was 
intended to address the issue of the power of simulation to teach job-related 
social skills to mildly handicapped young people, and to test the overall 
instructional effectiveness of the SUCCESS program. Due to lack of adequate 
funds, the formative evaluation was severely truncated and the summative 
evaluation activities have been deferred indefinitely. The following is a 

description of the formative and summative evaluation activities that had been 
planned, and a summary of the formative evaluation procedures that were 
implemented but not completed.' 

Original Formative Evaluation Plan 

The original plan for the formative evaluation of the project activities 
had three major components: (a) a description of the field sites and samples of 
handicapped students and their teachers, (b) a description of the evaluation 
procedures and instruments which were to be used to measure the appropriateness 
of the materials for the intended audiences, and (c) the procedures from the data 
gathering which would result in revisions and modifications. The target of the 
formative evaluation was on the instructional materials for the students and the 
teacher training materials which will accompany the student instructional 
materials. At the end of the formative evaluation process, and any revisions 
resulting from this process, there was to have been a final product which would be 
the focus of the summative evaluation. 
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The samples for the formative evaluation were intended to represent a 
wide range of handicapped learners from the natural laboratories used in this 
project. The population from which the samples were to be drawn consists of 
approximately 370 secondary students in a two large Denver area school districts 
who are identified as handicapped and who are enrolled in special education 
programs. Stratified random samples of these students were to be selected from 
each of the target populations (learning disabled, mildly retarded, emotionally 
disturbed). Teachers from these field sites were also to be included as evaluators 
in the formative stage. Commitments had been secured from the participating 
schools for their involvement in the development and evaluation process. 

The original focus of the formative evaluation was on determining 
whether the simulation materials could adequately teach the identified social 
skills (e.g., whether they promote skill mastery, transfer, and generalization). 
However, on the advice of the.project's .Review Board, the focus of the formative 
evaluation was shifted to address issues surrounding the implementation of the 
SUCCESS curriculum in school settings. In particular, questions to be addressed 
included student reactions to the material, ease of use and flexibility for 
teachers, adequacy and completeness of curriculum materials, and further insights 
into how the SUCCESS Program could be integrated with other ongoing programs. 

Original Summative Evaluation Plans 

The major purpose of the summative evaluation was to determine 
whether computer-based simulation materials could be effective in (a) facilitating 
the transfer and generalization of targeted job-related social skills and (b) 
enhancing perceived competence of job-related social skills. The effectiveness of 
these materials was to be examined relative to a "typical" social skills training 
program and to no social skills training. 

1^3 
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The evaluation design directly addressed the generalization and transfer 
issue in the construction of assessment conditions within two experimental groups 
and two control groups: 



• Experimental Group 1 (EGl) . This group would receive 
computer-based simulation training and be tested for 
generalization and transfer after completion of each of the 
first three levels of training. 

• Experimental Group 2 (EG2) . This group would receive 
computer-based simulation training, but not be tested for 
generalization or transfer after the completion of each of 
the first three levels of training. 

• Control Group 1 (CGI) . This group would receive a 
"typical" noncomputer-based job-related social skills 
program (e.g., Marathon). 

• Control Group 2 (CG2) . This group would not receive any 
specialized training in job-related social skills, but would 
engage in a "discussion group." 

The general design, including conditions, estimated sample ns, measures, 
and general points of data collection, is depicted in Figure 1. As can be seen in 
the figure, all groups were to be administered a pretest and a posttest that would 
include measures of perceived competence, skill mastery, and generalization and 
transfer. When any specific comparison between one experimental and one 
control group is considered, we planned to implement a quasi-experimental design 
typically termed a "nonequivalent control group design." This basic design 
possesses good internal validity, which we intended to enhance by instituting 
appropriate statistical controls. 

We included two experimental groups because we were interested in 
assessing the generalization and transfer of acquired job-related social skills to 
novel role-play situations at the completion of each level of the program (i.e., via 
EGl at the various posttest points), but wanted to guard against the charge that 
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such testing could be considered part of the treatment (i.e., by comparing EGl vs. 
EG2). Comparisons within EGl would have allowed for determinations of the 
contribution of each simulation program level (and its associated strategies) to 
generalization and transfer. 

The general hypotheses to be tested were that after completion of the 
program, the two experimental groups would have significantly higher perceived 
competence than either control group; the experimental groups would 
demonstrate significantly greater transfer of social skills than either control 
group; the two experimental groups would demonstrate significantly more 
generalization of social skills than control groups; and that there would be 
differences in results by type of handicapping condition. A number of other 
testable hypotheses were evident from the evaluation design and these were to be 
refined during the formative evaluation period. 

The population we intended to generalize included handicapped students 
ages 16 to 21 who demonstrate learning disabilities, mild mental retardation, or 
behavior disorders, and who are enrolled in a special education program designed 
to prepare them for job placement after high school. Subjects were to come from 
eight high schools in the Denver metropolitan area that were not involved in the 
formative evaluation. 

The specific battery of pretest and posttest measures was to include 
measures of perceived competence, such as Harter's (1986) Self-Perception 
Profile for Adolescents, mastery tests over all objectives in the first three 
simulation levels, and skill and role-play instruments to measure transfer and 
generalization. Instruments for Level U were to consist of the short role plays for 
measuring generalization and transfer. Level 5 measures and the follow-up 
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testing were to include an employer survey, rating scales, and interviews to assess 
job success. Also, we anticipated gathering additional contextual information 
from observational measures, anecdotal reports, and curriculum evaluation 
questionnaires. 

The first step in the data analysis would have been to generate simple 
descriptive statistics for the variable set (for each group and for the total 
sample): means, medians, standard deviations, ranges, zero-order correlations, 

and reliabilities, where appropriate. Such descriptive statistics would have 
provided an initial understanding of the data and would have served as a check on 
the accuracy of the data. Next, simple cross-tabulations were to be generated. 
For example, multiway frequency tables of demographic characteristics by group 
membership would have helped determine whether the experimental and control 
groups were comparable. While not sophisticated, the cross-tabulations would 
have .served two major. purposes: they.would have . been useful for presentation in 
nontechnical summaries where readers are not likely to be familiar with 
complicated statistical procedures and, they would have provided initial 
information concerning what relationships to examine in higher order analyses. 
We expected to employ a multiple regression approach in the latter stages of data 
analysis. 

Partial Formative Evaluation Conducted 

This evaluation has been designed to provide information on classroom 
implementation issues regarding effective individualized and group administration 
practices, and to provide feedback on any logistical or content concerns that 
should be addressed in final revisions of the materials. The evaluation was begun 
on February 3, 1988 with the training of four teachers and three administrators. 
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These individuals participated in a half-day training session in which they were 
introduced to the background and philosophy of the curriculum, received a 
demonstration and practice with the on-line and off-line Phase I materials, and 
were briefed on the purpose and procedures of the evaluation. Specifically, the 
information being collected in the formative evaluation of the Phase I materials 
consisted of the following: 



• Student Evaluations of the SUCCESS Material 

At the end of the tryout period (May 27, 1988), students 
were to be asked to fill out an evaluation form which 
asks them their reactions to (a) the clarity and 
completeness of directions they received from the 
computer; (b) how "real" the job-related situations 
seemed to them; (c) how "real" the people in the 
situations seemed to them; (d) what they learned from 
the lessons; (e) the difficulty level; (f) how interesting 
vs, boring the lessons were; (g) how interesting the 
characters were; (h) what they liked least and best about 
the computer lessons; and (i) how well they think other 
students would like these types of computer lessons, 

• Teacher Comments on the SUCCESS Material 

At the end of the tryout period (May 27, 1988), teachers 
were to be asked to make comments on an evaluation 
form with respect to (a) the usefulness and completeness 
of the Teacher Manual; (b) how well the SUCCESS 
material fits in the normal classroom activities; (c) how 
well the content of the material fits in; (d) their 
assessment of student reactions to the computer-based 
lessons and role-playing material; (e) their evaluations of 
the completeness of the role-playing directions and level 
of difficulty for students; (f) the kinds of questions 
students had about the lessons; (g) difficulties they 
encountered starting up the computer-based lessons and 
operating the equipment; (h) the kinds of students for 
which the materials seemed most appropriate; and (i) 
their comments and suggestions for improving the 
program. 
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• SUCCESS Program Trouble Log 

Throughout the formative field test of Phase I materials, 
teachers were asked to record any problems or "troubles" 
encountered with the operation of the computer-based 
lessons. They were provided with forms on which to 
record the date, nature of the problem, and how the 
problem was resolved. 

• Project Staff Observations and Monitoring 

Three observation and monitoring sessions at the Golden 
High School field test site were planned for project staff 
during the months of March, April, and May. These day- 
long sessions were intended to provide "third-party" data 
on student reactions to the SUCCESS material under 
group and individual conditions, and the logistics of 
teacher implementation of the program under these two 
conditions. These sessions would have also provided the 
project staff with opportunities to collect additional 
data to be used in cross-checking and verifying data 
provided by teachers and students, and to identify issues 
to be addressed at the debriefing session planned for 
project staff and teachers at the end of the Phase I field 
test (May 27, 1988). 

• Teacher Evaluations of Student Performance on Role- 
Playing 

As part of the Level 4 Role-Playing Exercises, teachers 
were asked to fill out checklists for each exercise in 
■ which they evaluate whether students are demonstrating 
required social skills components relevant to each 
exercise. Procedures for scoring these checklists were 
also provided so that an overall assessment of skill level 
could be obtained. 

At the midpoint of the Phase I formative evaluation (March 31, 1988), 
incomplete data in all of these categories were available. Data available for 
summarization primarily consists of teacher reports of trouble with the operation 
of the computer-based lessons and results of staff monitoring and observation in 
March. 
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Mid-Point Formative Evaluation Results 



As described in the first section of this report, by the end of March, 12 
learning disabled, nine mildly retarded, and nine emotionally disturbed students 
were participating in combinations of individual and group administrations of the 
Phase I materials. Within this group of students, seven of the learning disabled 
students participated in the group mode and five participated in an individualized 
mode; all nine mildly retarded students participated in an individualized mode; 
and the nine emotionally disturbed students participated in a group mode. 
Decisions regarding the group vs. individual mode were made on the basis of 
teachers' assessments of the type of learning problems students were experiencing 
and their abilities to handle the group vs. individual modes. 

For students in the group mode, they had completed nine of the 15 on- 
line lessons (i.e., the three introductory lessons, two Level 1 lessons each in 
Accepting Criticism and Asking for Help, and two Level 2 lessons in Accepting 
Criticism). Students participating in the individual mode had completed fewer of 
these lessons, with the mildly retarded students completing the fewest number of 
lessons. 

Data collected during the staff monitoring and observation session on 
March 15, 1988 consisted of the following: 

• Student reactions under group and individual conditions; 

• Logistics of teacher implementation of program under group and 
individual conditions; 

• Review of teacher "Trouble Reports". 

With respect to student reactions under the group condition, the nine 
emotionally disturbed students were observed during their group session. The 
teacher started the lesson for that day, stopping at points where a student 
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response was required. Students' attention varied during the presentation, with 
some being very attentive and others spending time "talking" with the student 
next to them. The teacher was very skillful in getting students on task and in 
relating the choices they needed to make during the lesson to choices they have to 
make regarding their classroom behavior. As the lesson progressed, students 
began to pay more attention and actively became involved in making the required 
choices and filling out their worksheets. Near the end of the class session, after 
students had completed the lesson, the teacher asked the project staff whether 
they would like to talk to individual students about their reactions so far to the 
SUCCESS program. At that point, five students volunteered to provide 
comments. In general, their comments indicated that they were enjoying the 
lessons and felt the information would help them on their jobs. Students varied in 
terms of whether they found the materials too easy vs. too difficult, but they 
generally agreed that the grade level for the materials was too low and that the 
information was something they should have had earlier in their schooling. In 
discussing these comments later with the teacher, he indicated that this was a 
"typical" reaction of these kinds of students in that they frequently state that 
activities are "too easy" and they should know the information already. He 
emphasized, however, that the conceptual and emotional problems present in this 
group of students were such that they needed the level of repetition and detail 
presented in the lessons. 

During the rest of the morning, three mildly retarded and one learning 
disabled student were observed participating in a lesson in an individualized mode. 
The teacher working with these students was adept at helping students who had 
difficulty with parts of the lesson or seemed confused. She allowed students to go 
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at their own pace and ask questions as necessary. For all students observed in the 
individual mode, attention to the lesson was very high. When students were 
interviewed by project staff at the end of the class period, the learning disabled 
student commented that she had had lots of jobs and found the materials very 
relevant. She particularly liked the combination of the visual and audio media as 
it helped her better understand the material. She did feel, however, that the 
lessons could be "harder" and present more difficult situations to figure out. On 
the other hand, the mildly retarded students generally commented that the 
materials were at the right level of difficulty and that they found the audio paired 
with visual materials to be particularly helpful in remembering and thinking about 
the new concepts and skills being presented. 

The two teachers who were working with the group and individualized 
modes, respectively, on the day of the monitoring session indicated that they were 
experiencing no difficulties with these two different presentation modes. They 
did feel, however, that it would be helpful to have a section of the Teacher's 
Manual that dealt specifically with how to use the program with students in the 
different handicapping conditions. In general, the teachers were very pleased and 
excited about the materials and their potential for helping students with not only 
job-related social skills, but also with developing more general social problem- 
solving skills that they could use in a variety of social situations. They had, on 
their own, come up with a number of strategies and supplemental materials that 
they felt were important in taking students beyond the job context in thinking 
about how to apply the general NOTICE, THINK, SAY and DO model. 

Finally, with respect to comments on the teachers' "Trouble Log," the 
only problem reported was an occasional difficulty with reading digital signals on 
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the audio tape. This specific problem is due to a characteristic of the prototype 
interface which is being used in the field test. This particular problem has been 
addressed in the manufactured version being produced by the publisher. 
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V. RECOMMENDATIONS 



Future Implementation of the SUCCESS Curriculum 

On the basis of preliminary formative evaluation activities completed to 
date, a number of areas in which recommendations for future implementations 
have been identified. These areas include the following: 

• Modifications to the Teacher Manual 

• Modifications to Student Off-line Materials 

• Group vs. Individualized Implementation Strategies 

With respect to the Teacher Manual, we have learned that teachers need 
guidance with structuring activities for student involvement in the on-line 
materials. That is, students in the three handicapping conditions attend, process, 
and retrieve information in highly individual and distinctive ways. Although 
special education teachers are generally familiar with these individual learning 
styles, they are less familiar with the impact of computer-based technologies and 
simulation approaches, in particular, on these styles. We believe, therefore, that 
the Teacher Manual could be enhanced by a special section that specifically 
outlines suggestions for individualizing the computer-based lessons to 
accommodate the special learning needs of individual categories of learning 
difficulties. 

Second, additional student off-line materials that teachers identified as 
helpful during the initial part of the Phase I formative evaluation were printed 
copies of all on-line questions in each lesson. We prepared a set of these 
questions which teachers then used in both the individual and group modes to 
assist students in attending to and remembering key concepts and skills. In the 
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group mode, teachers also found these questions to be helpful in stimulating group 
discussions. We thus recommend the inclusion of these questions in the set of off- 
line materials for each on-line lesson, with teacher guidelines for their use. 

Finally, with respect to individualized and group implementations of the 
on-line materials, our preliminary formative evaluation findings suggest that 
teachers are very knowledgeable about the types of students and instructional 
conditions in their environments that are most conducive to each respective 
mode. Teachers' knowledge and experience with particular students and resources 
available in their school settings thus make them the best judge of how to 
implement a curriculum of this type. We would, therefore, recommend that 
future implementations emphasize teachers' expertise in making individual vs. 
group implementation decisions, guided by more in-depth individualization , 
considerations as discussed above. 

Additional Research and Evaluation Activities 

An obvious need exists to complete formative evaluation and revision 
activities on the entire SUCCESS curriculum. Although our preliminary 
evaluation of Phase I materials has been positive with respect to implementation 
issues, additional issues are likely to surface once the entire curriculum has been 
implemented. For example, questions about how best to sequence and structure 
the Accepting Criticism and Asking for Help on-line lessons may emerge, as well 
as issues about how best to implement Level 4 and 5 off-line materials. All of 
these issues may further impact the types of revisions that are needed to enhance 
the effectiveness of student and teacher materials. Addressing these issues and 
completing final revisions of the curriculum is seen as a critical step before 
summative evaluation issues can be adequately addressed. 
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Once a final revised curriculum is available, the stage will be set to 
explore summative evaluation issues regarding the power of simulation as an 
instructional technology in enhancing the transfer and generalization of this social 
skills training program. Equally important however, is the opportunity with the 
SUCCESS curriculum to also evaluate instructional design and theory issues that 
are related to the effectiveness of this curriculum in promoting skill 
generalization and transfer. Specifically, the opportunity is present to separately 
evaluate the various on-line components (metacognitive, cognitive, affective) as 
well as the relative contribution of these components as compared with the off- 
line practice and transfer strategies (role playing, on-the-job support materials). 
The design presented in our summative evaluation plan (see Summative Evaluation 
Plan (Task 10) submitted to the OSEP on September 22, 1986) provides a means to 
address these issues and to assist us in a better understanding of those 
technological vs. instructional components of a job-related social skills 
intervention that are critical to learning handicapped students' transfer and 
generalization of social skills that can enhance their successful transition to the 
world of work. 
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